{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# NVTabular demo on RecSys2020 Challenge\n",
    "\n",
    "## Overview\n",
    "\n",
    "NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.  It provides a high level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS cuDF library.\n",
    "\n",
    "### RecSys2020 Challenge\n",
    "\n",
    "The [RecSys](https://recsys.acm.org/) conference is the leading data science conference for recommender systems and organizes an annual competiton in recommender systems. The [RecSys Challenge 2020](https://recsys-twitter.com/), hosted by Twitter, was about predicting interactions of ~200 mio. tweet-user pairs. NVIDIA's team scored 1st place. The team explained the solution in the [blogpost](https://medium.com/rapids-ai/winning-solution-of-recsys2020-challenge-gpu-accelerated-feature-engineering-and-training-for-cd67c5a87b1f) and published the code on [github](https://github.com/rapidsai/deeplearning/tree/main/RecSys2020).\n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "This notebook covers the end-2-end pipeline, from loading the original .tsv file to training model, with NVTabular, dask_cudf and XGBoost. We demonstrate multi-GPU support for NVTabular and new `nvt.ops`, implemented based on the success of our RecSys2020 solution.\n",
    "1. **NVTabular** to preprocess the original .tsv file.\n",
    "2. **dask_cudf** to split the preprocessed data into a training and validation set.\n",
    "3. **NVTabular** to create additional features with only **~70 lines of code**.\n",
    "4. **dask_cudf** / **XGBoost on GPU** to train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# External Dependencies\n",
    "import time\n",
    "import glob\n",
    "import gc\n",
    "\n",
    "import cupy\n",
    "import cupy as cp\n",
    "import cudf\n",
    "import rmm\n",
    "import dask\n",
    "import dask_cudf\n",
    "\n",
    "import numpy as np\n",
    "import nvtabular as nvt\n",
    "import xgboost as xgb\n",
    "\n",
    "from dask_cuda import LocalCUDACluster\n",
    "from dask.distributed import Client\n",
    "from dask.distributed import wait\n",
    "from dask.utils import parse_bytes\n",
    "from dask.delayed import delayed\n",
    "from nvtabular.utils import device_mem_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_total_start = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "BASE_DIR = '/raid/data/recsys2020/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initalize our local cuda cluster and assign the memory pool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://127.0.0.1:35807</li>\n",
       "  <li><b>Dashboard: </b><a href='http://127.0.0.1:8787/status' target='_blank'>http://127.0.0.1:8787/status</a></li>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>2</li>\n",
       "  <li><b>Cores: </b>2</li>\n",
       "  <li><b>Memory: </b>49.17 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://127.0.0.1:35807' processes=2 threads=2, memory=49.17 GB>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster = LocalCUDACluster(protocol=\"tcp\", \n",
    "                           rmm_pool_size=\"31GB\"\n",
    ")\n",
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tcp://127.0.0.1:37925': None, 'tcp://127.0.0.1:41431': None}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize RMM pool on ALL workers\n",
    "def _rmm_pool():\n",
    "    rmm.reinitialize(\n",
    "        pool_allocator=True,\n",
    "        initial_pool_size=None, # Use default size\n",
    "    )\n",
    "    \n",
    "client.run(_rmm_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define the data schema:\n",
    "- features are the column names in the original .tsv file. The .tsv file has no header.\n",
    "- cat_names are the categorical features, we want to use.\n",
    "- cont_names are the numerical features, we want to use.\n",
    "- label_name contains the target column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = [\n",
    "    'text_tokens',    ###############\n",
    "    'hashtags',       #Tweet Features\n",
    "    'tweet_id',       #\n",
    "    'media',          #\n",
    "    'links',          #\n",
    "    'domains',        #\n",
    "    'tweet_type',     #\n",
    "    'language',       #\n",
    "    'timestamp',      ###############\n",
    "    'a_user_id',              ###########################\n",
    "    'a_follower_count',       #Engaged With User Features\n",
    "    'a_following_count',      #\n",
    "    'a_is_verified',          #\n",
    "    'a_account_creation',     ###########################\n",
    "    'b_user_id',              #######################\n",
    "    'b_follower_count',       #Engaging User Features\n",
    "    'b_following_count',      #\n",
    "    'b_is_verified',          #\n",
    "    'b_account_creation',     #######################\n",
    "    'b_follows_a',    #################### Engagement Features\n",
    "    'reply',          #Target Reply\n",
    "    'retweet',        #Target Retweet    \n",
    "    'retweet_comment',#Target Retweet with comment\n",
    "    'like',           #Target Like\n",
    "                      ####################\n",
    "]\n",
    "\n",
    "cat_names = [ \n",
    "    'hashtags', \n",
    "    'tweet_id', \n",
    "    'media', \n",
    "    'links',\n",
    "    'domains', \n",
    "    'tweet_type', \n",
    "    'language', \n",
    "    'a_user_id',\n",
    "    'a_is_verified',\n",
    "    'b_user_id', \n",
    "    'b_is_verified', \n",
    "    'b_follows_a', \n",
    "\n",
    "]\n",
    "\n",
    "cont_names = [\n",
    "    'timestamp',\n",
    "    'a_follower_count', \n",
    "    'a_following_count',\n",
    "    'a_account_creation',\n",
    "    'b_follower_count',\n",
    "    'b_following_count',\n",
    "    'b_account_creation'\n",
    "]\n",
    "\n",
    "label_name = [\n",
    "    'reply', \n",
    "    'retweet', \n",
    "    'retweet_comment', \n",
    "    'like'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize our NVTabular workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc = nvt.Workflow(cat_names=cat_names, \n",
    "                    cont_names=cont_names, \n",
    "                    label_name=label_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define two helper function, we apply in our NVTabular workflow:\n",
    "1. splitmedia2 splits the entries in media by `\\t` and keeps only the first two values (if available),\n",
    "2. count_token counts the number of token in a column (e.g. how many hashtags are in a tweet),"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitmedia2(col):\n",
    "    if col.shape[0] == 0:\n",
    "        return(col)\n",
    "    else:\n",
    "        return(col.str.split('\\t')[0].fillna('') + '_' + col.str.split('\\t')[1].fillna(''))\n",
    "    \n",
    "def count_token(col,token):\n",
    "    not_null = col.isnull()==0\n",
    "    return ((col.str.count(token)+1)*not_null).fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize a nvt.Dataset. The engine is `csv` as the `.tsv` file has a similar structure. The `.tsv` file uses the special character `\\x01` to separate columns. There is no header in the file and we define column names with the parameter names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "trains_itrs = nvt.Dataset(BASE_DIR + 'training.tsv', \n",
    "                          header=None, \n",
    "                          names=features, \n",
    "                          engine='csv', \n",
    "                          sep='\\x01', \n",
    "                          part_size='1GB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define our preprocessing workflow.\n",
    "\n",
    "1. Extract weekday from the timestamp column.\n",
    "2. Count the number of tokens in the columns hashtags, domains, links.\n",
    "3. Fill in na/missing values in target columns + hashtags, domains, links.\n",
    "4. Change data type to uint32 to save memory/storage.\n",
    "5. Apply splitmedia function.\n",
    "6. Encode columns in continous integer to save memory. Some categorical columns contain long hashes of type String as values to preserve the privacy of the users (e.g. userId, language, etc.). Long hashes of type String requires signficant amount of memory to store. We encode/map the Strings to continous Integer to save significant memory.\n",
    "7. Change data type to uint32 to save memory/storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc.add_feature([\n",
    "    # Extract weekday from the timestamp column\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'wd',\n",
    "        f = lambda col, gdf: cudf.to_datetime(col, unit='s').dt.weekday,\n",
    "        columns = ['timestamp'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Count the number of tokens in the columns hashtags, domains, links\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name='count_t',\n",
    "        f=lambda col, gdf: count_token(col,'\\t'),\n",
    "        columns=['hashtags', 'domains', 'links'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Fill in na/missing values in target columns + hashtags, domains, links\n",
    "    nvt.ops.FillMissing(columns=label_name + ['hashtags', 'domains', 'links']),\n",
    "    # Change data type to uint32 to save memory/storage\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name='astypeint32',\n",
    "        f=lambda col, gdf: col.astype(np.uint32),\n",
    "        columns=label_name + ['timestamp',\n",
    "                              'a_follower_count', \n",
    "                              'a_following_count',\n",
    "                              'a_account_creation',\n",
    "                              'b_follower_count',\n",
    "                              'b_following_count',\n",
    "                              'b_account_creation'],\n",
    "        replace=True\n",
    "    ),\n",
    "    # Apply splitmedia function\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name='splitmedia',\n",
    "        f=lambda col, gdf: splitmedia2(col),\n",
    "        columns=['media'],\n",
    "        replace=True\n",
    "    ),\n",
    "    # Encode columns in continous integer to save memory\n",
    "    nvt.ops.Categorify(\n",
    "        columns=['media', 'language', 'tweet_type', 'tweet_id', \n",
    "                 'a_user_id', 'b_user_id', 'hashtags', 'domains', 'links']\n",
    "    ),\n",
    "    # Change data type to uint32 to save memory/storage\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name='astypeint32_2',\n",
    "        f=lambda col, gdf: col.astype(np.uint32),\n",
    "        replace=True,\n",
    "        columns=['media', 'language', 'tweet_type', 'tweet_id', \n",
    "                 'a_user_id', 'b_user_id', 'hashtags', 'domains', 'links']\n",
    "    )\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply the workflow and save the output to `preprocess/`,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/core/column/string.py:1992: UserWarning: `expand` parameter defatults to True.\n",
      "  warnings.warn(\"`expand` parameter defatults to True.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 40s, sys: 1min 15s, total: 3min 56s\n",
      "Wall time: 5min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "time_preproc_start = time.time()\n",
    "proc.apply(trains_itrs, record_stats=True, output_path=BASE_DIR + 'preprocess/')\n",
    "time_preproc = time.time()-time_preproc_start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_metadata\t part.2.parquet   part.31.parquet  part.43.parquet\n",
      "part.0.parquet\t part.20.parquet  part.32.parquet  part.44.parquet\n",
      "part.1.parquet\t part.21.parquet  part.33.parquet  part.45.parquet\n",
      "part.10.parquet  part.22.parquet  part.34.parquet  part.46.parquet\n",
      "part.11.parquet  part.23.parquet  part.35.parquet  part.47.parquet\n",
      "part.12.parquet  part.24.parquet  part.36.parquet  part.48.parquet\n",
      "part.13.parquet  part.25.parquet  part.37.parquet  part.49.parquet\n",
      "part.14.parquet  part.26.parquet  part.38.parquet  part.5.parquet\n",
      "part.15.parquet  part.27.parquet  part.39.parquet  part.50.parquet\n",
      "part.16.parquet  part.28.parquet  part.4.parquet   part.6.parquet\n",
      "part.17.parquet  part.29.parquet  part.40.parquet  part.7.parquet\n",
      "part.18.parquet  part.3.parquet   part.41.parquet  part.8.parquet\n",
      "part.19.parquet  part.30.parquet  part.42.parquet  part.9.parquet\n"
     ]
    }
   ],
   "source": [
    "!ls $BASE_DIR/preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the training data by time into a train and validation set. The first 5 days are train and the last 2 days are for validation. We use the weekday for it. The first day of the dataset is a Thursday (weekday id = 3) and the last day is Wednesday (weekday id = 2). Therefore, we split the weekday ids 1 and 2 into the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.61 s, sys: 458 ms, total: 8.07 s\n",
      "Wall time: 37.5 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "330"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "time_split_start = time.time()\n",
    "\n",
    "df = dask_cudf.read_parquet(BASE_DIR + 'preprocess/*.parquet')\n",
    "VALID_DOW = [1, 2]\n",
    "valid = df[df['timestamp_wd'].isin(VALID_DOW)].reset_index(drop=True)\n",
    "train = df[~df['timestamp_wd'].isin(VALID_DOW)].reset_index(drop=True)\n",
    "train = train.sort_values([\"b_user_id\", \"timestamp\"]).reset_index(drop=True)\n",
    "valid = valid.sort_values([\"b_user_id\", \"timestamp\"]).reset_index(drop=True)\n",
    "train.to_parquet(BASE_DIR + 'nv_train/')\n",
    "valid.to_parquet(BASE_DIR + 'nv_valid/')\n",
    "time_split = time.time()-time_split_start\n",
    "\n",
    "del train; del valid\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Feature Engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can apply the actual feature engineering. First, we define the data schema again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "CATEGORICAL_COLUMNS = [\n",
    "    'hashtags', 'tweet_id', 'media', 'links', 'domains', 'tweet_type', 'language', \n",
    "    'a_user_id', 'a_is_verified', \n",
    "    'b_user_id', 'b_is_verified', \n",
    "    'b_follows_a', 'timestamp_wd'\n",
    "]\n",
    "\n",
    "CONTINUOUS_COLUMNS = [\n",
    "    'timestamp', \n",
    "    'a_follower_count', 'a_following_count', \n",
    "    'b_follower_count', 'b_following_count',\n",
    "    'hashtags_count_t', 'domains_count_t', 'links_count_t'\n",
    "]\n",
    "LABEL_COLUMNS = [\n",
    "    'reply', 'retweet', 'retweet_comment', 'like'\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initalize the NVTabular workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc = nvt.Workflow(\n",
    "    cat_names=CATEGORICAL_COLUMNS,\n",
    "    cont_names=CONTINUOUS_COLUMNS,\n",
    "    label_name=LABEL_COLUMNS\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define our preprocessing workflow.\n",
    "\n",
    "1. Transform the target columns into boolean (0/1) targets.\n",
    "2. Apply TargetEncoding with kfold of 5 and smoothing of 20. TargetEncoding is explained in [here](https://medium.com/rapids-ai/target-encoding-with-rapids-cuml-do-more-with-your-categorical-data-8c762c79e784) and [here](https://github.com/rapidsai/deeplearning/blob/main/RecSys2020Tutorial/03_3_TargetEncoding.ipynb).\n",
    "3. CountEncode the columns *media*, *tweet_type*, *language*, *a_user_id*, *b_user_id*. CountEncoding is explained [here](https://github.com/rapidsai/deeplearning/blob/main/RecSys2020Tutorial/03_4_CountEncoding.ipynb).\n",
    "4. Calculate the ratio of *a_following_count/a_follower_count*.\n",
    "5. Calculate the ratio of *b_following_count/b_follower_count*.\n",
    "6. Transform timestamp to datetime type.\n",
    "7. Extract hour.\n",
    "8. Extract minute.\n",
    "9. Extract seconds.\n",
    "10. Convert columns to float.\n",
    "11. Difference encode *b_follower_count, b_following_count, language* grouped by *b_user_id*. DifferenceEncoding is explained [here](https://github.com/rapidsai/deeplearning/blob/main/RecSys2020Tutorial/05_2_TimeSeries_Differences.ipynb).\n",
    "12. Fill missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc.add_feature([\n",
    "    # Transform the target columns into boolean (0/1) targets\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'change',\n",
    "        f = lambda col, gdf: (col>0).astype('int8'),\n",
    "        columns = LABEL_COLUMNS,\n",
    "        replace=True\n",
    "    ),\n",
    "    # Apply TargetEncoding with kfold of 5 and smoothing of 20\n",
    "    nvt.ops.TargetEncoding(\n",
    "        cat_groups = ['media', \n",
    "                      'tweet_type', \n",
    "                      'language', \n",
    "                      'a_user_id', \n",
    "                      'b_user_id', \n",
    "                      ['domains','language','b_follows_a','tweet_type','media','a_is_verified']],\n",
    "        cont_target = LABEL_COLUMNS,\n",
    "        kfold = 5,\n",
    "        p_smooth = 20\n",
    "    ),\n",
    "    # CountEncode the columns media, tweet_type, language, a_user_id, b_user_id\n",
    "    nvt.ops.JoinGroupby(\n",
    "        columns = [\n",
    "            'media', \n",
    "            'tweet_type', \n",
    "            'language', \n",
    "            'a_user_id', \n",
    "            'b_user_id'\n",
    "        ]\n",
    "    ),\n",
    "    # Calc the ratio of a_following_count/a_follower_count\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'a_ff_rate',\n",
    "        f = lambda col, gdf: gdf['a_following_count']/gdf['a_follower_count'],\n",
    "        columns = ['a_following_count'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Calc the ratio of b_following_count/b_follower_count\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'b_ff_rate',\n",
    "        f = lambda col, gdf: gdf['b_following_count']/gdf['b_follower_count'],\n",
    "        columns = ['b_following_count'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Transform timestamp to datetime type\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'to_datetime',\n",
    "        f = lambda col, gdf: cudf.to_datetime(col.astype('int32'), unit='s'),\n",
    "        columns = ['timestamp'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Extract hour\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'to_hour',\n",
    "        f = lambda col, gdf: col.dt.hour,\n",
    "        columns = ['timestamp_to_datetime'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Extract minute\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'to_minute',\n",
    "        f = lambda col, gdf: col.dt.minute,\n",
    "        columns = ['timestamp_to_datetime'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Extract seconds\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'to_second',\n",
    "        f = lambda col, gdf: col.dt.second,\n",
    "        columns = ['timestamp_to_datetime'],\n",
    "        replace=False\n",
    "    ),\n",
    "    # Convert columns to float\n",
    "    nvt.ops.LambdaOp(\n",
    "        op_name = 'asfloat',\n",
    "        f = lambda col, gdf: col.astype('float32'),\n",
    "        columns = ['b_follower_count','b_following_count','language'],\n",
    "        replace=True\n",
    "    ),\n",
    "    # Difference encode b_follower_count, b_following_count, language grouped by b_user_id\n",
    "    nvt.ops.DifferenceLag(\n",
    "        'b_user_id', \n",
    "        columns=['b_follower_count','b_following_count','language'],\n",
    "        shift = [1, -1]\n",
    "    ),\n",
    "    # Fill missing values\n",
    "    nvt.ops.FillMissing(fill_val=0)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize the train and valid as NVTabular datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = nvt.Dataset(glob.glob(BASE_DIR + 'nv_train/*.parquet'), \n",
    "                            engine='parquet', \n",
    "                            part_size=\"4GB\")\n",
    "valid_dataset = nvt.Dataset(glob.glob(BASE_DIR + 'nv_valid/*.parquet'), \n",
    "                            engine='parquet', \n",
    "                            part_size=\"4GB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply the workflow to the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/core/join/join.py:368: UserWarning: can't safely cast column from right with type int64 to uint32, upcasting to int64\n",
      "  \"right\", dtype_r, dtype_l, libcudf_join_type\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 30s, sys: 2min 5s, total: 5min 36s\n",
      "Wall time: 5min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "time_fe_start = time.time()\n",
    "proc.apply(train_dataset, record_stats=True, output_path=BASE_DIR + 'nv_train_fe/')\n",
    "proc.apply(valid_dataset, record_stats=False, output_path=BASE_DIR + 'nv_valid_fe/')\n",
    "time_fe = time.time()-time_fe_start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the preprocessing and feature engineering is done, we can train a model to predict our targets. We load our datasets with `dask_cudf`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = dask_cudf.read_parquet(BASE_DIR + 'nv_train_fe/*.parquet')\n",
    "valid = dask_cudf.read_parquet(BASE_DIR + 'nv_valid_fe/*.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['timestamp', 'a_follower_count', 'a_following_count',\n",
       "       'b_follower_count', 'b_following_count', 'hashtags_count_t',\n",
       "       'domains_count_t', 'links_count_t', 'hashtags', 'tweet_id', 'media',\n",
       "       'links', 'domains', 'tweet_type', 'language', 'a_user_id',\n",
       "       'a_is_verified', 'b_user_id', 'b_is_verified', 'b_follows_a',\n",
       "       'timestamp_wd', 'reply', 'retweet', 'retweet_comment', 'like',\n",
       "       'TE_media_reply', 'TE_media_retweet', 'TE_media_retweet_comment',\n",
       "       'TE_media_like', 'TE_tweet_type_reply', 'TE_tweet_type_retweet',\n",
       "       'TE_tweet_type_retweet_comment', 'TE_tweet_type_like',\n",
       "       'TE_language_reply', 'TE_language_retweet',\n",
       "       'TE_language_retweet_comment', 'TE_language_like', 'TE_a_user_id_reply',\n",
       "       'TE_a_user_id_retweet', 'TE_a_user_id_retweet_comment',\n",
       "       'TE_a_user_id_like', 'TE_b_user_id_reply', 'TE_b_user_id_retweet',\n",
       "       'TE_b_user_id_retweet_comment', 'TE_b_user_id_like',\n",
       "       'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_reply',\n",
       "       'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet',\n",
       "       'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet_comment',\n",
       "       'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_like',\n",
       "       'media_count', 'tweet_type_count', 'language_count', 'a_user_id_count',\n",
       "       'b_user_id_count', 'a_following_count_a_ff_rate',\n",
       "       'b_following_count_b_ff_rate', 'timestamp_to_datetime',\n",
       "       'timestamp_to_datetime_to_hour', 'timestamp_to_datetime_to_minute',\n",
       "       'timestamp_to_datetime_to_second', 'b_follower_count_DifferenceLag_1',\n",
       "       'b_following_count_DifferenceLag_1', 'language_DifferenceLag_1',\n",
       "       'b_follower_count_DifferenceLag_-1',\n",
       "       'b_following_count_DifferenceLag_-1', 'language_DifferenceLag_-1'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use following input features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['media', 'tweet_type', 'language', 'a_follower_count', 'a_following_count', 'a_is_verified', 'b_follower_count', 'b_following_count', 'b_is_verified', 'b_follows_a', 'hashtags_count_t', 'domains_count_t', 'links_count_t', 'TE_media_reply', 'TE_media_retweet', 'TE_media_retweet_comment', 'TE_media_like', 'TE_tweet_type_reply', 'TE_tweet_type_retweet', 'TE_tweet_type_retweet_comment', 'TE_tweet_type_like', 'TE_language_reply', 'TE_language_retweet', 'TE_language_retweet_comment', 'TE_language_like', 'TE_a_user_id_reply', 'TE_a_user_id_retweet', 'TE_a_user_id_retweet_comment', 'TE_a_user_id_like', 'TE_b_user_id_reply', 'TE_b_user_id_retweet', 'TE_b_user_id_retweet_comment', 'TE_b_user_id_like', 'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_reply', 'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet', 'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet_comment', 'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_like', 'media_count', 'tweet_type_count', 'language_count', 'a_user_id_count', 'b_user_id_count', 'b_follower_count_DifferenceLag_1', 'b_following_count_DifferenceLag_1', 'language_DifferenceLag_1', 'b_follower_count_DifferenceLag_-1', 'b_following_count_DifferenceLag_-1', 'language_DifferenceLag_-1', 'timestamp_to_datetime_to_hour', 'timestamp_to_datetime_to_minute', 'timestamp_to_datetime_to_second'] 51\n"
     ]
    }
   ],
   "source": [
    "features = [\n",
    "    'media', 'tweet_type', 'language',\n",
    "    'a_follower_count', 'a_following_count', 'a_is_verified', \n",
    "    'b_follower_count', 'b_following_count', 'b_is_verified', 'b_follows_a',\n",
    "    'hashtags_count_t', 'domains_count_t', 'links_count_t',\n",
    "    'TE_media_reply', 'TE_media_retweet', 'TE_media_retweet_comment', 'TE_media_like',\n",
    "    'TE_tweet_type_reply', 'TE_tweet_type_retweet',\n",
    "    'TE_tweet_type_retweet_comment', 'TE_tweet_type_like',\n",
    "    'TE_language_reply', 'TE_language_retweet',\n",
    "    'TE_language_retweet_comment', 'TE_language_like', 'TE_a_user_id_reply',\n",
    "    'TE_a_user_id_retweet', 'TE_a_user_id_retweet_comment',\n",
    "    'TE_a_user_id_like', 'TE_b_user_id_reply', 'TE_b_user_id_retweet',\n",
    "    'TE_b_user_id_retweet_comment', 'TE_b_user_id_like',\n",
    "    'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_reply',\n",
    "    'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet',\n",
    "    'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_retweet_comment',\n",
    "    'TE_domains_language_b_follows_a_tweet_type_media_a_is_verified_like',\n",
    "    'media_count', 'tweet_type_count', 'language_count', 'a_user_id_count', 'b_user_id_count',\n",
    "    'b_follower_count_DifferenceLag_1', 'b_following_count_DifferenceLag_1', 'language_DifferenceLag_1',\n",
    "    'b_follower_count_DifferenceLag_-1', 'b_following_count_DifferenceLag_-1', 'language_DifferenceLag_-1',\n",
    "    'dt_dow', 'timestamp_to_datetime_to_hour', \n",
    "    'timestamp_to_datetime_to_minute', 'timestamp_to_datetime_to_second'\n",
    "]\n",
    "features = [x for x in features if x in train.columns]\n",
    "label_names = ['reply', 'retweet', 'retweet_comment', 'like']\n",
    "other = ['tweet_id']\n",
    "print(features, len(features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop the columns, which are not required for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DoneAndNotDoneFutures(done=set(), not_done=set())"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RMV = [x for x in train.columns if not(x in features+label_names+other)]\n",
    "\n",
    "train = train.drop(RMV,axis=1)\n",
    "RMV = [c for c in RMV if c in valid.columns]\n",
    "valid = valid.drop(RMV,axis=1)    \n",
    "wait(train)\n",
    "wait(valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our experiments show that we require only 10% of the training dataset. Our feature engineering, such as TargetEncoding, uses the training datasets and leverage the information of the full dataset. The actual model training does not require all the data, anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71043477\n",
      "7103396\n",
      "Using 51 features: 51\n"
     ]
    }
   ],
   "source": [
    "SAMPLE_RATIO = 0.1\n",
    "SEED = 1\n",
    "\n",
    "if SAMPLE_RATIO < 1.0:\n",
    "    train['sample'] = train['tweet_id'].map_partitions(lambda cudf_df: cudf_df.hash_encode(stop=10))\n",
    "    print(len(train))\n",
    "    \n",
    "    train = train[train['sample']<10*SAMPLE_RATIO]\n",
    "    train, = dask.persist(train)\n",
    "    print(len(train))\n",
    "\n",
    "\n",
    "Y_train = train[label_names]\n",
    "Y_train, = dask.persist(Y_train)    \n",
    "    \n",
    "train = train.drop(['sample','tweet_id']+label_names,axis=1)\n",
    "train, = dask.persist(train)\n",
    "\n",
    "print('Using %i features:'%(len(features)),train.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the training dataset, our experiments show that 35% of our validation dataset is enough to get a good estimate of the performance metric. 35% of the validation dataset has a similar size as the test set of the RecSys2020 competition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27142542\n",
      "10842998\n"
     ]
    }
   ],
   "source": [
    "SAMPLE_RATIO = 0.35\n",
    "SEED = 1\n",
    "if SAMPLE_RATIO < 1.0:\n",
    "    print(len(valid))\n",
    "    valid['sample'] = valid['tweet_id'].map_partitions(lambda cudf_df: cudf_df.hash_encode(stop=10))\n",
    "    \n",
    "    valid = valid[valid['sample']<10*SAMPLE_RATIO]\n",
    "    valid, = dask.persist(valid)\n",
    "    print(len(valid))\n",
    "    \n",
    "Y_valid = valid[label_names]\n",
    "Y_valid, = dask.persist(Y_valid)\n",
    "\n",
    "valid = valid.drop(['sample','tweet_id']+label_names,axis=1)\n",
    "valid, = dask.persist(valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize our XGBoost parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XGB Version 1.2.0-SNAPSHOT\n"
     ]
    }
   ],
   "source": [
    "print('XGB Version',xgb.__version__)\n",
    "\n",
    "xgb_parms = { \n",
    "    'max_depth':8, \n",
    "    'learning_rate':0.1, \n",
    "    'subsample':0.8,\n",
    "    'colsample_bytree':0.3, \n",
    "    'eval_metric':'logloss',\n",
    "    'objective':'binary:logistic',\n",
    "    'tree_method':'gpu_hist',\n",
    "    'predictor' : 'gpu_predictor'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no dup :) \n",
      "X_train.shape (Delayed('int-e22ee0aa-be07-40f1-a0c8-629154e58409'), 51)\n",
      "X_valid.shape (Delayed('int-767f21d5-d6d2-49dc-8321-71d0b45639f7'), 51)\n"
     ]
    }
   ],
   "source": [
    "if train.columns.duplicated().sum()>0:\n",
    "    raise Exception(f'duplicated!: { train.columns[train.columns.duplicated()] }')\n",
    "print('no dup :) ')\n",
    "print(f'X_train.shape {train.shape}')\n",
    "print(f'X_valid.shape {valid.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We optimize the datatype a last time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in train.columns:\n",
    "    if train[col].dtype=='bool':\n",
    "        train[col] = train[col].astype('int8')\n",
    "        valid[col] = valid[col].astype('int8')\n",
    "train,valid = dask.persist(train,valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train our XGBoost models. The challenge requires to predict 4 targets, does a user\n",
    "1. like a tweet\n",
    "2. reply a tweet\n",
    "3. comment a tweet\n",
    "4. comment and reply a tweet\n",
    "<br><br>\n",
    "We train 4x XGBoost models for 300 rounds on a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#########################\n",
      "### reply\n",
      "#########################\n",
      "Creating DMatrix...\n",
      "Took 0.1 seconds\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py:3518: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited\n",
      "  self.sync(self._update_scheduler_info)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 13.2 seconds\n",
      "Predicting...\n",
      "Took 0.4 seconds\n",
      "#########################\n",
      "### retweet\n",
      "#########################\n",
      "Creating DMatrix...\n",
      "Took 0.1 seconds\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py:3518: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited\n",
      "  self.sync(self._update_scheduler_info)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 13.3 seconds\n",
      "Predicting...\n",
      "Took 0.4 seconds\n",
      "#########################\n",
      "### retweet_comment\n",
      "#########################\n",
      "Creating DMatrix...\n",
      "Took 0.1 seconds\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py:3518: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited\n",
      "  self.sync(self._update_scheduler_info)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 12.3 seconds\n",
      "Predicting...\n",
      "Took 0.3 seconds\n",
      "#########################\n",
      "### like\n",
      "#########################\n",
      "Creating DMatrix...\n",
      "Took 0.2 seconds\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py:3518: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited\n",
      "  self.sync(self._update_scheduler_info)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 13.7 seconds\n",
      "Predicting...\n",
      "Took 0.4 seconds\n",
      "CPU times: user 3.5 s, sys: 358 ms, total: 3.86 s\n",
      "Wall time: 54.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "time_train_start = time.time()\n",
    "\n",
    "NROUND = 300\n",
    "VERBOSE_EVAL = 50    \n",
    "oof = np.zeros((len(valid),len(label_names)))\n",
    "preds = []\n",
    "for i in range(4):\n",
    "\n",
    "    name = label_names[i]\n",
    "    print('#'*25);print('###',name);print('#'*25)\n",
    "       \n",
    "    start = time.time(); print('Creating DMatrix...')\n",
    "    dtrain = xgb.dask.DaskDMatrix(client,data=train,label=Y_train.iloc[:, i])\n",
    "    print('Took %.1f seconds'%(time.time()-start))\n",
    "             \n",
    "    start = time.time(); print('Training...')\n",
    "    model = xgb.dask.train(client, xgb_parms, \n",
    "                           dtrain=dtrain,\n",
    "                           num_boost_round=NROUND,\n",
    "                           verbose_eval=VERBOSE_EVAL) \n",
    "    print('Took %.1f seconds'%(time.time()-start))\n",
    "        \n",
    "    start = time.time(); print('Predicting...')\n",
    "    preds.append(xgb.dask.predict(client,model,valid))\n",
    "    print('Took %.1f seconds'%(time.time()-start))\n",
    "        \n",
    "    del model, dtrain\n",
    "\n",
    "time_train = time.time()-time_train_start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "yvalid = Y_valid[label_names].values.compute()\n",
    "oof = cupy.array([i.values.compute() for i in preds]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hosts of the RecSys2020 competition provide code for calculating the performance metric `PRAUC` and `RCE`. We optimized the code to speed up the calculation, as well. Using cuDF / cupy, we calculate the performance metric on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import auc\n",
    "\n",
    "def precision_recall_curve(y_true,y_pred):\n",
    "    y_true = y_true.astype('float32')\n",
    "    ids = cupy.argsort(-y_pred) \n",
    "    y_true = y_true[ids]\n",
    "    y_pred = y_pred[ids]\n",
    "    y_pred = cupy.flip(y_pred,axis=0)\n",
    "\n",
    "    acc_one = cupy.cumsum(y_true)\n",
    "    sum_one = cupy.sum(y_true)\n",
    "    \n",
    "    precision = cupy.flip(acc_one/cupy.cumsum(cupy.ones(len(y_true))),axis=0)\n",
    "    precision[:-1] = precision[1:]\n",
    "    precision[-1] = 1.\n",
    "\n",
    "    recall = cupy.flip(acc_one/sum_one,axis=0)\n",
    "    recall[:-1] = recall[1:]\n",
    "    recall[-1] = 0\n",
    "    n = (recall==1).sum()\n",
    "    \n",
    "    return precision[n-1:],recall[n-1:],y_pred[n:]\n",
    "\n",
    "def compute_prauc(pred, gt):\n",
    "    prec, recall, thresh = precision_recall_curve(gt, pred)\n",
    "    recall, prec = cupy.asnumpy(recall), cupy.asnumpy(prec)\n",
    "    prauc = auc(recall, prec)\n",
    "    return prauc\n",
    "\n",
    "def log_loss(y_true,y_pred,eps=1e-7, normalize=True, sample_weight=None):\n",
    "    y_true = y_true.astype('int32')\n",
    "    y_pred = cupy.clip(y_pred, eps, 1 - eps)\n",
    "    if y_pred.ndim == 1:\n",
    "        y_pred = cupy.expand_dims(y_pred, axis=1)\n",
    "    if y_pred.shape[1] == 1:\n",
    "        y_pred = cupy.hstack([1 - y_pred, y_pred])\n",
    "\n",
    "    y_pred /= cupy.sum(y_pred, axis=1, keepdims=True)\n",
    "    loss = -cupy.log(y_pred)[cupy.arange(y_pred.shape[0]), y_true]\n",
    "    return _weighted_sum(loss, sample_weight, normalize).item()\n",
    "\n",
    "def _weighted_sum(sample_score, sample_weight, normalize):\n",
    "    if normalize:\n",
    "        return cupy.average(sample_score, weights=sample_weight)\n",
    "    elif sample_weight is not None:\n",
    "        return cupy.dot(sample_score, sample_weight)\n",
    "    else:\n",
    "        return sample_score.sum()\n",
    "\n",
    "def compute_rce_fast(pred, gt):\n",
    "    cross_entropy = log_loss(gt, pred)\n",
    "    yt = np.mean(gt)     \n",
    "    strawman_cross_entropy = -(yt*np.log(yt) + (1 - yt)*np.log(1 - yt))\n",
    "    return (1.0 - cross_entropy/strawman_cross_entropy)*100.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we calculate the performance metric PRAUC and RCE for each target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reply                PRAUC:0.14059 RCE:17.00633\n",
      "retweet              PRAUC:0.52260 RCE:28.32939\n",
      "retweet_comment      PRAUC:0.05128 RCE:10.76598\n",
      "like                 PRAUC:0.76884 RCE:23.83179\n"
     ]
    }
   ],
   "source": [
    "txt = ''\n",
    "for i in range(4):\n",
    "    prauc = compute_prauc(oof[:,i], yvalid[:, i])#.item()\n",
    "    rce   = compute_rce_fast(oof[:,i], yvalid[:, i]).item()\n",
    "    txt_ = f\"{label_names[i]:20} PRAUC:{prauc:.5f} RCE:{rce:.5f}\"\n",
    "    print(txt_)\n",
    "    txt += txt_ + '\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_total = time.time()-time_total_start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 823.59s\n",
      "\n",
      "1. Preprocessing:       318.75s\n",
      "2. Splitting:           37.41s\n",
      "3. Feature engineering: 337.98s\n",
      "4. Training:            54.91s\n"
     ]
    }
   ],
   "source": [
    "print('Total time: {:.2f}s'.format(time_total))\n",
    "print()\n",
    "print('1. Preprocessing:       {:.2f}s'.format(time_preproc))\n",
    "print('2. Splitting:           {:.2f}s'.format(time_split))\n",
    "print('3. Feature engineering: {:.2f}s'.format(time_fe))\n",
    "print('4. Training:            {:.2f}s'.format(time_train))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
